<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>Test-driven development with Twisted</title>
</head>

<body>

<h1>Test-driven development with Twisted</h1>

<p>Writing good code is hard, or at least it can be. A major challenge is
to ensure that your code remains correct as you add new functionality.</p>

<p><a href="http://en.wikipedia.org/wiki/Unit_test">Unit testing</a> is a
modern, light-weight testing methodology in widespread use in many
programming languages. Development that relies on unit tests is often
referred to as Test-Driven Development 
(<a href="http://en.wikipedia.org/wiki/Test-driven_development">TDD</a>). 
Most Twisted code is tested using TDD.</p>

<p>To gain a solid understanding of unit testing in Python, you should read
the <a href="http://docs.python.org/library/unittest.html">unittest --
Unit testing framework chapter</a> of the <a
href="http://docs.python.org/library/index.html">Python Library
Reference</a>. There is also a ton of information available online and in
books.</p>

<h2>Introductory example of Python unit testing</h2>

<p>This document is principally a guide to Trial, Twisted's unit testing
framework. Trial is based on Python's unit testing framework. While we do not
aim to give a comprehensive guide to general Python unit testing, it will be
helpful to consider a simple non-networked example before expanding to cover a
networking code that requires the special capabilities of Trial. If you are
already familiar with unit test in Python, jump straight to the section
specific to <a href="#twisted">testing Twisted code</a>.</p>

<p><div class="note">In what follows we will make a series of refinements
to some simple classes. In order to keep the examples and source code links
complete and to allow you to run Trial on the intermediate results at every
stage, I add <code>_N</code> (where the <code>N</code> are successive
integers) to file names to keep them separate. This is a minor visual
distraction that should be ignored.</div></p>

<h2>Creating an API and writing tests</h2>

<p>We'll create a library for arithmetic calculation. First, create a
project structure with a directory called <code
class="shell">calculus</code> containing an empty <code
class="py-filename">__init__.py</code> file.</p>

<p>Then put the following simple class definition API into <code
class="py-filename">calculus/base_1.py</code>:</p>

<a href="listings/trial/calculus/base_1.py" class="py-listing">base_1.py</a>

<p>(Ignore the <code class="python">test-case-name</code> comment for
now. You'll see why that's useful <a href="#comment">below</a>.)</p>

<p>We've written the interface, but not the code. Now we'll write a set of
tests. At this point of development, we'll be expecting all tests to
fail. Don't worry, that's part of the point. Once we have a test framework
functioning, and we have some decent tests written (and failing!), we'll go
and do the actual development of our calculation API. This is the preferred
way to work for many people using TDD - write tests first, make sure they
fail, then do development. Others are not so strict and write tests after
doing the development.</p>

<p>Create a <code class="shell">test</code> directory beneath <code
class="shell">calculus</code>, with an empty <code
class="py-filename">__init__.py</code> file. In a <code
class="py-filename">calculus/test/test_base_1.py</code>, put the
following:</p>

<a href="listings/trial/calculus/test/test_base_1.py" class="py-listing">test_base_1.py</a>

<p>You should now have the following 4 files:

<pre class="shell">
    calculus/__init__.py
    calculus/base_1.py
    calculus/test/__init__.py
    calculus/test/test_base_1.py
</pre>
</p>

<p>To run the tests, there are two things you must get set up. Make sure
you get these both done - nothing below will work unless you do.</p>

<p>First, make sure that the directory that <em>contains</em> your
 <code class="shell">calculus</code> directory is in your Python load path. If you're
using the Bash shell on some form of unix (e.g., Linux, Mac OS X), run
 <code class="shell">PYTHONPATH=&quot;$PYTHONPATH:`pwd`/..&quot;</code> at
the command line in the <code class="shell">calculus</code> directory. Once you have your
Python path set up correctly, you should be able to run Python from the
command line and <code class="python">import calculus</code> without seeing
an import error.</p>

<p>Second, make sure you can run the <code class="shell">trial</code>
command. That is, make sure the directory containing the <code class="shell">trial</code>
program on you system is in your shell's <code
class="shell">PATH</code>. The easiest way to check if you have this is to
try running <code class="shell">trial --help</code> at the command line. If
you see a list of invocation options, you're in business. If your shell
reports something like <code class="shell">trial: command not found</code>,
make sure you have Twisted installed properly, and that the Twisted
 <code class="shell">bin</code> directory is in your <code class="shell">PATH</code>. If
you don't know how to do this, get some local help, or figure it out by
searching online for information on setting and changing environment
variables for you operating system.</p>

<p>With those (one-time) preliminary steps out of the way, let's perform
the tests. Run <code class="shell">trial calculus.test.test_base_1</code> from the
command line when you are in the directory containing the <code class="shell">calculus</code>
directory.

You should see the following output (though your files are probably not in
 <code class="shell">/tmp</code>:</p>

<pre class="shell">
$ trial calculus.test.test_base_1
calculus.test.test_base_1
  CalculationTestCase
    test_add ...                                                         [FAIL]
    test_divide ...                                                      [FAIL]
    test_multiply ...                                                    [FAIL]
    test_subtract ...                                                    [FAIL]

===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/tmp/calculus/test/test_base_1.py", line 8, in test_add
    self.assertEqual(result, 11)
twisted.trial.unittest.FailTest: not equal:
a = None
b = 11


calculus.test.test_base_1.CalculationTestCase.test_add
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/tmp/calculus/test/test_base_1.py", line 23, in test_divide
    self.assertEqual(result, 2)
twisted.trial.unittest.FailTest: not equal:
a = None
b = 2


calculus.test.test_base_1.CalculationTestCase.test_divide
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/tmp/calculus/test/test_base_1.py", line 18, in test_multiply
    self.assertEqual(result, 60)
twisted.trial.unittest.FailTest: not equal:
a = None
b = 60


calculus.test.test_base_1.CalculationTestCase.test_multiply
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/tmp/calculus/test/test_base_1.py", line 13, in test_subtract
    self.assertEqual(result, 4)
twisted.trial.unittest.FailTest: not equal:
a = None
b = 4


calculus.test.test_base_1.CalculationTestCase.test_subtract
-------------------------------------------------------------------------------
Ran 4 tests in 0.042s

FAILED (failures=4)
</pre>

<p>How to interpret this output? You get a list of the individual tests, each
followed by its result. By default, failures are printed at the end, but this
can be changed with the <code class="shell">-e</code> (or <code
class="shell">--rterrors</code>) option.</p>

<p>One very useful thing in this output is the fully-qualified name of the
failed tests. This appears at the bottom of each =-delimited area of the
output. This allows you to copy and paste it to just run a single test you're
interested in. In our example, you could run <code class="shell">trial
calculus.test.test_base_1.CalculationTestCase.test_subtract</code> from the
shell.</p>

<p>Note that trial can use different reporters to modify its output. Run
 <code class="shell">trial --help-reporters</code> to see a list of
reporters.</p>

<p>
The tests can be run by <code class="shell">trial</code> in multiple ways:
<ul>
  <li><code class="shell">trial calculus</code>: run all the tests for the
  calculus package.</li>

  <li><code class="shell">trial calculus.test</code>: run using Python's
  <code class="python">import</code> notation.</li>

  <li><code class="shell">trial calculus.test.test_base_1</code>: as above, for
  a specific test module. You can follow that logic by putting your class name
  and even a method name to only run those specific tests.</li>

  <li><a name="comment"></a><code class="shell">trial
  --testmodule=calculus/base_1.py</code>: use the <code
  class="python">test-case-name</code> comment in the first line of
  <code class="py-filename">calculus/base_1.py</code> to find the tests.</li>

  <li><code class="shell">trial calculus/test</code>: run all the tests in the
  test directory (not recommended).</li>

  <li><code class="shell">trial calculus/test/test_base_1.py</code>: run a
  specific test file (not recommended).</li>
</ul>

The first 3 versions using full qualified names are strongly encouraged: they
are much more reliable and they allow you to easily be more selective in your
test runs.
</p>

<p>You'll notice that Trial create a <code class="shell">_trial_temp</code> directory in
the directory where you run the tests. This has a file called
 <code class="shell">test.log</code> which contains the log output of the tests (created
using <code class="python">log.msg</code> or <code
class="python">log.err</code> functions). Examine this file if you add
logging to your tests.</p>

<h2>Making the tests pass</h2>

<p>Now that we have a working test framework in place, and our tests are
failing (as expected) we can go and try to implement the correct API. We'll do
that in a new version of the above base_1
module, <code class="py-filename">calculus/base_2.py</code>:</p>

<a href="listings/trial/calculus/base_2.py" class="py-listing">base_2.py</a>

<p>We'll also create a new version of test_base_1 which imports and tests this
new implementation,
in <code class="py-filename">calculus/test_base_2.py</code>:</p>

<p><a href="listings/trial/calculus/test/test_base_2.py" class="py-listing">test_base_2</a> is a copy of test_base_1, but with the import changed. Run <code
class="shell">trial</code> again as above, and your tests should now pass:</p>

<pre class="shell">
$ trial calculus.test.test_base_2

Running 4 tests.
calculus.test.test_base
  CalculationTestCase
    test_add ...                                                           [OK]
    test_divide ...                                                        [OK]
    test_multiply ...                                                      [OK]
    test_subtract ...                                                      [OK]

-------------------------------------------------------------------------------
Ran 4 tests in 0.067s

PASSED (successes=4)
</pre>

<h3>Factoring out common test logic</h3>

<p>You'll notice that our test file contains redundant code. Let's get rid
of that. Python's unit testing framework allows your test class to define a
 <code class="python">setUp</code> method that is called before
 <em>each</em> test method in the class. This allows you to add attributes
to <code class="python">self</code> that can be used in tests
methods. We'll also add a parameterized test method to further simplify the
code.</p>

<p>Note that a test class may also provide the counterpart of <code
class="python">setUp</code>, named <code class="python">tearDown</code>,
which will be called after <em>each</em> test (whether successful or
not). <code class="python">tearDown</code> is mainly used for post-test
cleanup purposes. We will not use <code class="python">tearDown</code>
until later.</p>

<p>Create <code class="py-filename">calculus/test/test_base_2b.py</code> as
follows:</p>

<a href="listings/trial/calculus/test/test_base_2b.py" class="py-listing">test_base_2b.py</a>

<p>Much cleaner, no?</p>

<p>We'll now add some additional error tests. Testing just for successful
use of the API is generally not enough, especially if you expect your code
to be used by others. Let's make sure the <code
class="python">Calculation</code> class raises exceptions if someone tries
to call its methods with arguments that cannot be converted to
integers.</p>

<p>We arrive at <code
class="py-filename">calculus/test/test_base_3.py</code>:</p>

<a href="listings/trial/calculus/test/test_base_3.py" class="py-listing">test_base_3.py</a>

<p>We've added four new tests and one general-purpose function, <code
class="python">_test_error</code>. This function uses the <code
class="python">assertRaises</code> method, which takes an exception class,
a function to run and its arguments, and checks that calling the function
on the arguments does indeed raise the given exception.</p>

<p>If you run the above, you'll see that not all tests fail. In Python it's
often valid to add and multiply objects of different and even differing
types, so the code in the add and mutiply tests does not raise an exception
and those tests therefore fail. So let's add explicit type conversion to
our API class. This brings us to <code
class="py-filename">calculus/base_3.py</code>:</p>

<a href="listings/trial/calculus/base_3.py" class="py-listing">base_3.py</a>

<p>Here the <code class="python">_make_ints</code> helper function tries to
convert a list into a list of equivalent integers, and raises a <code
class="python">TypeError</code> in case the conversion goes wrong.

<div class="note">The <code class="python">int</code> conversion can also
raise a <code class="python">TypeError</code> if passed something of the
wrong type, such as a list. We'll just let that exception go by as <code
class="python">TypeError</code> is already what we want in case something
goes wrong.</div></p>


<a name="twisted"></a>
<h2>Twisted specific testing</h2>

<p>Up to this point we've been doing fairly standard Python unit testing.
With only a few cosmetic changes (most importantly, directly importing
 <code class="python">unittest</code> instead of using Twisted's <code
class="API" base="twisted.trial">unittest</code> version) we could make the
above tests run using Python's standard library unit testing framework.</p>

<p>Here we will assume a basic familiarity with Twisted's network I/O, timing,
and Deferred APIs.  If you haven't already read them, you should read the
documentation on <a href="servers.xhtml">Writing
Servers</a>, <a href="clients.xhtml">Writing Clients</a>,
and <a href="defer.xhtml">Deferreds</a>.</p>

<p>Now we'll get to the real point of this tutorial and take advantage of
Trial to test Twisted code.</p>

<h2>Testing a protocol</h2>

<p>We'll now create a custom protocol to invoke our class from within a
telnet-like session. We'll remotely call commands with arguments and read back
the response. The goal will be to test our network code without creating
sockets.</p>

<h3>Creating and testing the server</h3>

<p>First we'll write the tests, and then explain what they do.  The first
version of the remote test code is:</p>

<a href="listings/trial/calculus/test/test_remote_1.py" class="py-listing">test_remote_1.py</a>

<p>To fully understand this client, it helps a lot to be comfortable with
the Factory/Protocol/Transport pattern used in Twisted.</p>

<p>We first create a protocol factory object. Note that we have yet to see
the <code class="python">RemoteCalculationFactory</code> class. It is in
 <code class="py-filename">calculus/remote_1.py</code> below. We
call <code class="python">buildProtocol</code> to ask the factory to build us a
protocol object that knows how to talk to our server.  We then make a fake
network transport, an instance of <code class="python">twisted.test.proto_helpers.StringTransport</code>
class (note that test packages are generally not part of Twisted's public API;
<code class="python">twisted.test.proto_helpers</code> is an exception).  This fake
transport is the key to the communications. It is used to emulate a network
connection without a network. The address and port passed to <code>buildProtocol</code>
are typically used by the factory to choose to immediately deny remote connections; since we're using a fake transport, we can choose any value that will be acceptable to the factory. In this case the factory just ignores the address, so we don't need to pick anything in particular.</p>

<p>Testing protocols without the use of real network connections is both simple and recommended when testing Twisted
code.  Even though there are many tests in Twisted that use the network,
most good tests don't. The problem with unit tests and networking is that
networks aren't reliable. We cannot know that they will exhibit reasonable
behavior all the time. This creates intermittent test failures due to
network vagaries. Right now we're trying to test our Twisted code, not
network reliability.  By setting up and using a fake transport, we can
write 100% reliable tests. We can also test network failures in a deterministic manner, another important part of your complete test suite.</p>

<p>The final key to understanding this client code is the <code
class="python">_test</code> method. The call to <code
class="python">dataReceived</code> simulates data arriving on the network
transport. But where does it arrive? It's handed to the <code
class="python">lineReceived</code> method of the protocol instance (in
 <code class="py-filename">calculus/remote_1.py</code> below). So the client
is essentially tricking the server into thinking it has received the
operation and the arguments over the network. The server (once again, see
below) hands the work off to its <code
class="python">CalculationProxy</code> object which in turn hands it to its
 <code class="python">Calculation</code> instance. The result is written
back via <code class="python">sendLine</code> (into the fake string
transport object), and is then immediately available to the client, who
fetches it with <code class="python">tr.value()</code> and checks that it
has the expected value. So there's quite a lot going on behind the scenes
in the two-line <code class="python">_test</code> method above.</p>

<p><em>Finally</em>, let's see the implementation of this protocol. Put the
following into <code class="py-filename">calculus/remote_1.py</code>:</p>

<a href="listings/trial/calculus/remote_1.py" class="py-listing">remote_1.py</a>

<p>As mentioned, this server creates a protocol that inherits from <code
class="API" base="twisted.protocols">basic.LineReceiver</code>, and then a
factory that uses it as protocol. The only trick is the <code
class="python">CalculationProxy</code> object, which calls <code
class="python">Calculation</code> methods through <code
class="python">remote_*</code> methods. This pattern is used frequently in
Twisted, because it is very explicit about what methods you are making
accessible.</p>

<p>If you run this test (<code class="shell">trial
calculus.test.test_remote_1</code>), everything should be fine. You can also
run a server to test it with a telnet client. To do that, call <code
class="shell">python calculus/remote_1.py</code>. You should have the following output:</p>

<pre class="shell">
2008-04-25 10:53:27+0200 [-] Log opened.
2008-04-25 10:53:27+0200 [-] __main__.RemoteCalculationFactory starting on 46194
2008-04-25 10:53:27+0200 [-] Starting factory &lt;__main__.RemoteCalculationFactory instance at 0x846a0cc&gt;
</pre>

<p>46194 is replaced by a random port. You can then call telnet on it:</p>
<pre>
$ telnet localhost 46194
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
add 4123 9423
13546
</pre>

<p>It works!</p>

<h3>Creating and testing the client</h3>

<p>Of course, what we build is not particulary useful for now : we'll now build
a client to our server, to be able to use it inside a Python program. And it
will serve our next purpose.</p>

<p>Create <code
class="py-filename">calculus/test/test_client_1.py</code>:</p>

<a href="listings/trial/calculus/test/test_client_1.py" class="py-listing">test_client_1.py</a>

<p>It's really symmetric to the server test cases. The only tricky part is
that we don't use a client factory. We're lazy, and it's not very useful in
the client part, so we instantiate the protocol directly.</p>

<p>Incidentally, we have introduced a very important concept here: the tests
now return a Deferred object, and the assertion is done in a callback. When
a test returns a Deferred, the reactor is run until the Deferred fires and
its callbacks run.  The
important thing to do here is to <strong>not forget to return the
Deferred</strong>. If you do, your tests will pass even if nothing is asserted.
That's also why it's important to make tests fail first: if your tests pass
whereas you know they shouldn't, there is a problem in your tests.</p>

<p>We'll now add the remote client class to produce <code
class="py-filename">calculus/client_1.py</code>:</p>

<a href="listings/trial/calculus/client_1.py" class="py-listing">client_1.py</a>


<h2>More good practices</h2>

<h3>Testing scheduling</h3>

<p>When testing code that involves the passage of time, waiting e.g. for a two hour timeout to occur in a test is not very realistic. Twisted provides a solution to this, the <code class="API" base="twisted.internet.task">Clock</code> class that allows one to simulate the passage of time.</p>

<p>As an example we'll test the code for client request timeout: since our client
uses TCP it can hang for a long time (firewall, connectivity problems, etc...).
So generally we need to implement timeouts on the client side. Basically it's
just that we send a request, don't receive a response and expect a timeout error
to be triggered after a certain duration.
</p>

<a href="listings/trial/calculus/test/test_client_2.py" class="py-listing">test_client_2.py</a>

<p>What happens here? We instantiate our protocol as usual, the only trick
is to create the clock, and assign <code class="python">proto.callLater</code> to
 <code class="python">clock.callLater</code>. Thus, every callLater calls in the protocol
will finish before <code class="python">clock.advance()</code> returns.</p>

<p>In the new test (test_timeout), we call <code
class="python">clock.advance</code>, that simulates and advance in time
(logically it's similar to a <code class="python">time.sleep</code> call). And
we just have to verify that our Deferred got a timeout error.</p>

<p>Let's implement that in our code.</p>

<a href="listings/trial/calculus/client_2.py" class="py-listing">client_2.py</a>

<p>The only important thing here is to not forget to cancel our callLater
when everything went fine.</p>

<h3>Cleaning up after tests</h3>

<p>This chapter is mainly intended for people that want to have sockets or
processes created in their tests. If it's still not obvious, you must try to
avoid that like the plague, because it ends up with a lot of problems, one of
them being intermittent failures. And intermittent failures are the plague
of automated tests.</p>

<p>To actually test that, we'll launch a server with our protocol.</p>

<a href="listings/trial/calculus/test/test_remote_2.py"
class="py-listing">test_remote_2.py</a>

<p>Recent versions of trial will fail loudly if you remove the
 <code class="python">stopListening</code> call, which is good.</p>

<p>Also, you should be aware that <code class="python">tearDown</code> will
called in any case, after success or failure. So don't expect that every
objects you created in the test method are present, because your tests may
have failed in the middle.</p>

<p>Trial also has a <code class="python">addCleanup</code> method, which makes
these kind of cleanups easy and removes the need for <code class="python">tearDown
</code>. For example, you could remove the code in <code class="python">_test</code> 
this way:</p>

<pre class="python">
def setUp(self):
    factory = RemoteCalculationFactory()
    self.port = reactor.listenTCP(0, factory, interface="127.0.0.1")
    self.addCleanup(self.port.stopListening)

def _test(self, op, a, b, expected):
    creator = protocol.ClientCreator(reactor, RemoteCalculationClient)
    def cb(client):
        self.addCleanup(self.client.transport.loseConnection)
        return getattr(client, op)(a, b).addCallback(self.assertEqual, expected)
    return creator.connectTCP('127.0.0.1', self.port.getHost().port).addCallback(cb)
</pre>

<p>This remove the need of a tearDown method, and you don't have to check for
the value of self.client: you only call addCleanup when the client is
created.</p>

<h3>Handling logged errors</h3>

<p>Currently, if you send an invalid command or invalid arguments to our
server, it logs an exception and closes the connection. This is a perfectly
valid behavior, but for the sake of this tutorial, we want to return an error
to the user if he sends invalid operators, and log any errors on server side. 
So we'll want a test like this:</p>

<pre class="python">
def test_invalidParameters(self):
    self.proto.dataReceived('add foo bar\r\n')
    self.assertEqual(self.tr.value(), "error\r\n")
</pre>

<a href="listings/trial/calculus/remote_2.py" class="py-listing">remote_2.py</a>

<p>If you try something like that, it will not work. Here is the output you should have:</p>

<pre class="shell">
trial calculus.test.test_remote_3.RemoteCalculationTestCase.test_invalidParameters
calculus.test.test_remote_3
  RemoteCalculationTestCase
    test_invalidParameters ...                                          [ERROR]

===============================================================================
[ERROR]: calculus.test.test_remote_3.RemoteCalculationTestCase.test_invalidParameters

Traceback (most recent call last):
  File "/tmp/calculus/remote_2.py", line 27, in lineReceived
    result = op(a, b)
  File "/tmp/calculus/base_3.py", line 11, in add
    a, b = self._make_ints(a, b)
  File "/tmp/calculus/base_3.py", line 8, in _make_ints
    raise TypeError
exceptions.TypeError:
-------------------------------------------------------------------------------
Ran 1 tests in 0.004s

FAILED (errors=1)
</pre>

<p>At first, you could think there is a problem, because you catch this
exception. But in fact trial doesn't let you do that without controlling it:
you must expect logged errors and clean them. To do that, you have to use the
 <code class="python">flushLoggedErrors</code> method. You call it with the
exception you expect, and it returns the list of exceptions logged since the
start of the test. Generally, you'll want to check that this list has the
expected length, or possibly that each exception has an expected message. We do
the former in our test:</p>

<a href="listings/trial/calculus/test/test_remote_3.py"
class="py-listing">test_remote_3.py</a>

<h2>Resolve a bug</h2>

<p>A bug was left over during the development of the timeout (probably several
bugs, but that's not the point), concerning the reuse of the protocol when you
got a timeout: the connection is not dropped, so you can get timeout forever.
Generally an user will come to you saying &quot;I have this strange problem on
my crappy network environment. It seems you could solve it with doing XXX at
YYY.&quot;</p>

<p>Actually, this bug can be corrected several ways. But if you correct it 
without adding tests, one day you'll face a big problem: regression. 
So the first step is adding a failing test.</p>

<a href="listings/trial/calculus/test/test_client_3.py" class="py-listing">test_client_3.py</a>
<p>What have we done here ?
<ul>
    <li>We switched to StringTransportWithDisconnection. This transport manages
    <code class="python">loseConnection</code> and forwards it to its protocol.</li>
    <li>We assign the protocol to the transport via the <code class="python">protocol 
   </code> attribute.</li>
    <li>We check that after a timeout our connection has closed.</li>
</ul>
</p>

<p>For doing that, we then use the <code class="python">TimeoutMixin</code>
class, that does almost everything we want. The great thing is that it almost
changes nothing to our class.</p>

<a href="listings/trial/calculus/client_3.py" class="py-listing">client_3.py</a>

<h2>Testing Deferreds without the reactor</h2>

<p>Above we learned about returning Deferreds from test methods in order to make
assertions about their results, or side-effects that only happen after they
fire.  This can be useful, but we don't actually need the feature in this
example.  Because we were careful to use <code class="python">Clock</code>, we
don't need the global reactor to run in our tests.  Instead of returning the
Deferred with a callback attached to it which performs the necessary assertions,
we can use a testing helper,
 <code class="API" base="twisted.trial.unittest.SynchronousTestCase">successResultOf</code> (and
the corresponding error-case helper
 <code class="API" base="twisted.trial.unittest.SynchronousTestCase">failureResultOf</code>), to
extract its result and make assertions against it directly.  Compared to
returning a Deferred, this avoids the problem of forgetting to return the
Deferred, improves the stack trace reported when the assertion fails, and avoids
the complexity of using global reactor (which, for example, may then require
cleanup).</p>

<a href="listings/trial/calculus/test/test_client_4.py" class="py-listing">test_client_4.py</a>

<p>This version of the code makes the same assertions, but no longer returns any
Deferreds from any test methods.  Instead of making assertions about the result
of the Deferred in a callback, it makes the assertions as soon as
it <em>knows</em> the Deferred is supposed to have a result (in
the <code>_test</code> method and in <code>test_timeout</code>
and <code>test_timeoutConnectionLost</code>).  The possibility
of <em>knowing</em> exactly when a Deferred is supposed to have a test is what
makes <code>successResultOf</code> useful in unit testing, but prevents it from being
applicable to non-testing purposes.</p>

<p><code>successResultOf</code> will raise an exception (failing the test) if
the <code>Deferred</code> passed to it does not have a result, or has a failure
result.  Similarly, <code>failureResultOf</code> will raise an exception (also
failing the test) if the <code>Deferred</code> passed to it does not have a
result, or has a success result.  There is a third helper method for testing the
final case,
 <code class="API" base="twisted.trial.unittest.SynchronousTestCase">assertNoResult</code>,
which only raises an exception (failing the test) if the <code>Deferred</code> passed
to it <em>has</em> a result (either success or failure).</p>

<h2>Dropping into a debugger</h2>

<p>In the course of writing and running your tests, it is often helpful to
employ the use of a debugger. This can be particularly helpful in tracking down
where the source of a troublesome bug is in your code. Python's standard library
includes a debugger in the form of the
<a href="http://docs.python.org/library/pdb.html">pdb</a> module.
Running your tests with <code class="python">pdb</code> is as simple as invoking
twisted with the <code class="shell">--debug</code> option, which will start
<code class="python">pdb</code> at the beginning of the execution of your test
suite.</p>

<p>Trial also provides a <code class="shell">--debugger</code> option which can
run your test suite using another debugger instead. To specify a debugger other
than <code class="python">pdb</code>, pass in the fully-qualified name of an
object that provides the same interface as <code class="python">pdb</code>.
Most third-party debuggers tend to implement an interface similar to
<code class="python">pdb</code>, or at least provide a wrapper object that
does. For example, invoking trial with the line
<code class="shell">trial --debug --debugger pudb</code> will open the
<a href="http://pypi.python.org/pypi/pudb">PuDB</a> debugger instead, provided
it is properly installed.</p>

<h2>Code coverage</h2>

<p>Code coverage is one of the aspects of software testing that shows how much
your tests cross (cover) the code of your program. There are different kind of
measures: path coverage, condition coverage, statement coverage... We'll only
consider statement coverage here, whether a line has been executed or not.
</p>

<p>Trial has an option to generate the statement coverage of your tests.
This option is --coverage. It creates a coverage directory in _trial_temp,
with a file .cover for every modules used during the tests. The ones
interesting for us are calculus.base.cover and calculus.remote.cover.  In
front of each line is the number of times you went through during the
tests, or the marker '&gt;&gt;&gt;&gt;&gt;&gt;' if the line was not
covered. If you went through all the tutorial to this point, you should
have complete coverage :).</p>

<p>Again, this is only another useful pointer, but it doesn't mean your
code is perfect: your tests should consider every possibile input and
output, to get <strong>full</strong> coverage (condition, path, etc.) as well
.</p>

<h2>Conclusion</h2>

<p>So what did you learn in this document?
<ul>
    <li>How to use the trial command-line tool to run your tests</li>
    <li>How to use string transports to test individual clients and servers
    without creating sockets</li>
    <li>If you really want to create sockets, how to cleanly do it so that it
    doesn't have bad side effects</li>
    <li>And some small tips you can't live without.</li>
</ul>
If one of the topics still looks cloudy to you, please give us your feedback!
You can file tickets to improve this document
<a href="http://twistedmatrix.com/">on the Twisted web site</a>.
</p>

</body>
</html>
